{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb2052b55e485eb2",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "In this tutorial, we show how to use an open-source embedding model and LLM on BentoCloud with vector database on Zilliz Cloud to build a RAG (Retrieval Augmented Generation) application. Specifically, you will do the following:\n",
    "\n",
    "- Generating vector embeddings with open-source models with BentoML or BentoCloud.\n",
    "- Inserting your data into a vector database for RAG\n",
    "  - Creating your Zilliz Cloud vector database\n",
    "  - Parsing and embedding your data for insertion\n",
    "- Setting up your RAG application with an open-source LLM such as Llama 3 or Mistral on BentoCloud\n",
    "  - Composing prompt for LLM with context retrieved from the Zilliz Cloud vector database\n",
    "  - Generating a final answer\n",
    "\n",
    "BentoCloud is an AI Inference Platform for fast-moving AI teams, offering fully-managed infrastructure tailored for model inference. It works in conjunction with BentoML, an open-source model serving framework, to facilitate the easy creation and deployment of high-performance model services. Zilliz Cloud is fully-managed service for open-source vector database - Milvus, with flexible pricing and ease of management. You can sign-up for free on [BentoCloud](https://cloud.bentoml.com/signup) and [Zilliz Cloud](https://cloud.zilliz.com/signup?utm_source=partner&utm_medium=referral&utm_campaign=2024-01-18_product_zcp-demos_github&utm_content=bento). Later we will use the API keys from both services to finish the demo.\n",
    "\n",
    "We can interact with deployed BentoML Services in `Deployments`, and the corresponding END_POINT and API are located in `Playground` -> `Python`. For Zilliz Cloud vector db, we can access `URI` and `API` in `Cluster Details`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57be2275f305ab8",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Access BentoML and corresponding END_POINT and API:\n",
    "\n",
    "<img src=\"../../images/bento_zilliz_rag_p1.png\">\n",
    "\n",
    "Access Zilliz Cloud and corresponding URI and API:\n",
    "\n",
    "<img src=\"../../images/bento_zilliz_rag_p2.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d87a5eba4d95c0e3",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ! pip install bentoml>=1.2.12 pymilvus>=2.3.7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5331ef59a46c1285",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Serving Embeddings with BentoML/BentoCloud\n",
    "\n",
    "With BentoCloud, it's easy to spin up an embedding service by choosing one from the `Explore Models` page. For example, `sentence_transformers` is a popular embedding model and we will use it in our demo. Simply follow the above screenshot to get the API endpoint and token from the UI.\n",
    "\n",
    "Alternatively, if you prefer running the embedding model locally, you can use the same model served through BentoML using its [Sentence Transformers Embeddings](https://github.com/bentoml/BentoSentenceTransformers) repository. By running `service.py` file, it spins up a local server and assigns an API endpoint on it. Within the API endpoint, e.g. `http://localhost:3000`, it is loading the `all-MiniLM-L6-v2` Sentence Transformer model from Hugging Face and will use it to create embeddings.\n",
    "\n",
    "To use this endpoint, the idea is the same: just import `bentoml` and set up an HTTP client using the `SyncHTTPClient` by specifying the endpoint and optionally the token (only if you turn on `Endpoint Authorization` on BentoCloud)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5fdf73bb6bd74e85",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:52:51.474421Z",
     "start_time": "2024-04-22T19:52:50.233126Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bentoml\n",
    "\n",
    "BENTO_EMBEDDING_MODEL_END_POINT = \"BENTO_EMBEDDING_MODEL_END_POINT\"\n",
    "BENTO_API_TOKEN = \"BENTO_API_TOKEN\"\n",
    "\n",
    "embedding_client = bentoml.SyncHTTPClient(BENTO_EMBEDDING_MODEL_END_POINT, token=BENTO_API_TOKEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e61ed31c7cd162a",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Once we connect to the embedding_client, we can create a function that gets a list of embeddings from a list of strings. It's usually more efficient if the model inference is done in batch, thus why we group 25 text strings in each embedding request. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342c0720e7bd2446",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "After splitting the string list into batches, we call the `embedding_client` we created above to encode these sentence into vectors (this process is typically called \"embedding\"). The BentoML client returns a list of vectors — effectively a list of float number arrays. We ungroup them and put them in a flat list of vectors to prepare for insertion.\n",
    "\n",
    "If there are not more than 25 strings in the list of texts, we simply call the encode method from the client on the passed-in list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a514c16b9b167b35",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:52:54.340636Z",
     "start_time": "2024-04-22T19:52:54.307326Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_embeddings(texts: list) -> list:\n",
    "   if len(texts) > 25:\n",
    "       splits = [texts[x:x+25] for x in range(0, len(texts), 25)]\n",
    "       embeddings = []\n",
    "       for split in splits:\n",
    "           embedding_split = embedding_client.encode(\n",
    "               sentences = split\n",
    "           )\n",
    "           embeddings += embedding_split\n",
    "       return embeddings\n",
    "   return embedding_client.encode(\n",
    "       sentences=texts,\n",
    "   )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f86c58a46fb97e56",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Inserting Your Data into a Vector Database for Retrieval \n",
    "\n",
    "With our embedding function prepared, we can insert the vectors together with metadata into Zilliz Cloud for vector search later. The first step in this section is to start a client by connecting to Zilliz Cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e9325ac4980ae81",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "For this part, we simply import the `MilvusClient` module and initialize a Zilliz Cloud client that connect to your vector database in Zilliz Cloud or Milvus. If you have a self-hosted Milvus instance, you can reuse the code by simplying replacing the URI and Token with your Milvus credentials as Zilliz Cloud and Milvus share the exact same API. In the following part, we will use the Zilliz Cloud as an example to do the rest of vector database operations. The code block below also defines two constants: a collection name and the dimension. You can make up whatever collection name you want. The dimension size comes from the size of the embedding model, e.g. the Sentence Transformer model `all-MiniLM-L6-v2` produces vectors of 384 dimension. You can get the dimension spec by looking up the description of the embedding models on resources such as HuggingFace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f682a2ba84a2288c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:52:57.435194Z",
     "start_time": "2024-04-22T19:52:55.778527Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pymilvus import MilvusClient\n",
    "\n",
    "COLLECTION_NAME = \"Bento_Zilliz_RAG\" # random name for your collection\n",
    "DIMENSION = 384\n",
    "\n",
    "ZILLIZ_ENDPOINT = \"YOUR_ZILLIZ_ENDPOINT\"\n",
    "ZILLIZ_TOKEN = \"YOUR_ZILLIZ_TOKEN\"\n",
    "\n",
    "zilliz_client = MilvusClient(\n",
    "    uri=ZILLIZ_ENDPOINT,\n",
    "    token=ZILLIZ_TOKEN\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e973fd94f968df0",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Creating Your Zilliz Cloud Collection\n",
    "\n",
    "Creating a collection on Zilliz Cloud involves two steps: first, defining the schema, and second, defining the index. For this section, we need one module: `DataType` tells us what type of data will be in a field. We also need to use two functions to create schema and add fields. `create_schema()`:  creates a collection schema, `add_field`: adds a field to the schema of a collection.\n",
    "\n",
    "We can define the entire schema for the collection here. Or, we can simply define the two necessary pieces: `id` and `embedding`. Then, when it comes time to define the schema, we pass a parameter, `enabled_dynamic_field`, that lets us insert whatever fields we want as long as we also have the `id` and `embedding` fields. This lets us treat inserting data into Zilliz Cloud the same way we would treat a NoSQL database like MongoDB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "4415d32f22375a01",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:41:18.148767Z",
     "start_time": "2024-04-22T19:41:18.143478Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'auto_id': True, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'embedding', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 384}}], 'enable_dynamic_field': True}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pymilvus import MilvusClient, DataType, Collection\n",
    "\n",
    "# Create schema\n",
    "schema = MilvusClient.create_schema(\n",
    "    auto_id=True,\n",
    "    enable_dynamic_field=True,\n",
    ")\n",
    "\n",
    "# 3.2. Add fields to schema\n",
    "schema.add_field(field_name=\"id\", datatype=DataType.INT64, is_primary=True)\n",
    "schema.add_field(field_name=\"embedding\", datatype=DataType.FLOAT_VECTOR, dim=DIMENSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc94a82965838c09",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now that we have created our schema and successfully defined data field, we need to define the index. In terms of search, an \"index\" defines how we are going to map our data out for retrieval. We use the default choice [AUTOINDEX](https://docs.zilliz.com/docs/autoindex-explained) to index our data for this project. We also need to define how we’re going to measure vector distance. In this example, we use the COSINE.\n",
    "\n",
    "If you have advanced use case which needs to specify a specific index type, there are many index types offered in Zilliz Cloud such as IVF and HNSW.\n",
    "\n",
    "Once the index is defined, we create the index on the vector field — in this case, `embedding`.\n",
    "\n",
    "Next, we simply create the collection with the previously given name, schema and index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "f37e02ff8e12373d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:41:36.223714Z",
     "start_time": "2024-04-22T19:41:32.506707Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# prepare index parameters\n",
    "index_params = zilliz_client.prepare_index_params()\n",
    "\n",
    "# add index\n",
    "index_params.add_index(\n",
    "    field_name=\"embedding\", \n",
    "    index_type=\"AUTOINDEX\", # use autoindex instead of other complex indexing method\n",
    "    metric_type=\"COSINE\", # L2, COSINE, or IP\n",
    ")\n",
    "\n",
    "# create collection\n",
    "zilliz_client.create_collection(\n",
    "    collection_name=COLLECTION_NAME,\n",
    "    schema=schema,\n",
    "    index_params=index_params\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0d8dc2c52b2fec2",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Parsing and Embedding Your Data for Insertion\n",
    "With Zilliz Cloud ready and the connection made, we can insert data into our vector database. But, we have to get the data ready to insert first. For this example, we have a bunch of `txt` files available in the data folder of the repo. We split this data into chunks, embed it and store it in Zilliz Cloud.\n",
    "\n",
    "The text are usually too long for an embedding model to take as input. Let’s start by creating a function that chunks this text up. There are many ways to do chunking, but we’ll do this naively by splitting it at every new line. It returns the newly created list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "da149bb17c90fc7a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:53:00.494426Z",
     "start_time": "2024-04-22T19:53:00.460549Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# naively chunk on newlines\n",
    "def chunk_text(filename: str) -> list:\n",
    "   with open(filename, \"r\") as f:\n",
    "       text = f.read()\n",
    "   sentences = text.split(\"\\n\")\n",
    "   return sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29cdb8dceec00f00",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Next, we process each of the files we have. We get a list of all the file names and create an empty list to hold the chunked information. Then, we loop through all the files and run the above function on each to get a naive chunking of each file.\n",
    "\n",
    "Before we store the chunks, we need to clean them. If you look at how an individual file is chunked, you’ll see many empty lines, and we don’t want empty lines. Some lines are just tabs or other special characters. To avoid those, we create an empty list and store only the chunks above a certain length. For simplicity, we can use seven characters.\n",
    "\n",
    "Once we have a cleaned list of chunks from each document, we can store our data. We create a dictionary that maps each list of chunks to the document’s name — in this case, the city name. Then, we append all of these to the empty list we made above. This is saying that we will have a dictionary that contain a mapping between each city and their corresponding clean text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3594bf9e072e2f17",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "# please upload your data directory under this file's folder\n",
    "cities = os.listdir(\"city_data\")\n",
    "# store chunked text for each of the cities in a list of dicts\n",
    "city_chunks = []\n",
    "for city in cities:\n",
    "   chunked = chunk_text(f\"city_data/{city}\")\n",
    "   cleaned = []\n",
    "   for chunk in chunked:\n",
    "       if len(chunk) > 7:\n",
    "           cleaned.append(chunk)\n",
    "   mapped = {\n",
    "       \"city_name\": city.split(\".\")[0],\n",
    "       \"chunks\": cleaned\n",
    "   }\n",
    "   city_chunks.append(mapped)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a392ab74a405dddb",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "With a set of chunked texts for each city ready to go, it’s time to get some embeddings. Zilliz Cloud can take a list of dictionaries to insert into a collection so we can start with another empty list. For each of the dictionaries we created above, we need to get a list of embeddings to match the list of sentences.\n",
    "\n",
    "We do this by directly calling the `get_embeddings` function we created in the section using BentoML on each of the list of chunks. Now, we need to match them up. Since the list embeddings and the list of sentences should match by index, we can `enumerate` through either list to match them up.\n",
    "\n",
    "We match them up by creating a dictionary representing a single entry into Zilliz Cloud. Each entry includes the embedding, the related sentence and the city. It’s optional to include the city, but let’s include it so we can use it. Notice there’s no need to include an `id` in this entry. That’s because we chose to auto-increment the `id` when we made the schema above.\n",
    "\n",
    "We add each of these entries to the list as we loop through them. At the end, we have a list of dictionaries with each dictionary representing a single-row entry to Zilliz Cloud. We can then simply insert these entries into our Zilliz Cloud collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80aedfbdd915a290",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T19:59:06.743391Z",
     "start_time": "2024-04-22T19:57:49.480484Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "entries = []\n",
    "for city_dict in city_chunks:\n",
    "    # No need for the embeddings list if get_embeddings already returns a list of lists\n",
    "    embedding_list = get_embeddings(city_dict[\"chunks\"])  # returns a list of lists\n",
    "    # Now match texts with embeddings and city name\n",
    "    for i, embedding in enumerate(embedding_list):\n",
    "        entry = {\n",
    "            \"embedding\": embedding,\n",
    "            \"sentence\": city_dict[\"chunks\"][i],  # Assume \"chunks\" has the corresponding texts for the embeddings\n",
    "            \"city\": city_dict[\"city_name\"]\n",
    "        }\n",
    "        entries.append(entry)\n",
    "    print(entries)\n",
    "\n",
    "# Outside of the loop, now you upsert all the entries at once\n",
    "zilliz_client.insert(collection_name=COLLECTION_NAME,data=entries)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc3df6c5818ac8e",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Set up Your LLM for RAG\n",
    "To build a RAG app, we need to deploy an LLM on BentoCloud. Let’s use the latest Llama3 LLM. Once it is up and running, simply copy the endpoint and token of this model service and set up a client for it.\n",
    "\n",
    "\n",
    "<img src=\"../../images/bento_zilliz_rag_llm.jpeg\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "251fa5a06748ebc2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T20:20:25.702043Z",
     "start_time": "2024-04-22T20:20:25.036615Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "BENTO_LLM_END_POINT = \"BENTO_LLM_END_POINT\"\n",
    "\n",
    "llm_client = bentoml.SyncHTTPClient(BENTO_LLM_END_POINT, token=BENTO_API_TOKEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4449446f071c9ec",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Giving the LLM Instructions\n",
    "There are two things the LLM needs to know to do RAG: the question and the context. We can pass both of these at once by creating a function that takes two strings: the question and the context.\n",
    "\n",
    "Using this function, we use the BentoML client’s chat completion to call an LLM. For this example, we use the `llama2-7b-chat` model that already implemented by the BentoML.\n",
    "\n",
    "We give this model two “messages” that indicate how it should behave. First, we give a message to the LLM to tell it that it is answering a question from the user based solely on the given context. Next, we tell it that there will be a user, and we simply pass in the question.\n",
    "\n",
    "The other parameters are for tuning the model behavior. We can control the maximum number of tokens the model can produce.\n",
    "\n",
    "The function then returns the output from the client in a string format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "fceac560894bc10",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T20:20:35.345208Z",
     "start_time": "2024-04-22T20:20:35.319278Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def dorag(question: str, context: str):\n",
    "    \n",
    "    prompt = (f\"You are a helpful assistant. The user has a question. Answer the user question based only on the context: {context}. \\n\"\n",
    "              f\"The user question is {question}\")\n",
    "    \n",
    "    results = llm_client.generate(\n",
    "        max_tokens=1024,\n",
    "        prompt=prompt,\n",
    "    )\n",
    "    \n",
    "    res = \"\"\n",
    "    for result in results:\n",
    "        res += result\n",
    "    \n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b34426ab8fef0617",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## A RAG Example\n",
    "Now we’re ready. It’s time to ask a question. We can probably do this without creating a function, but making a function makes it nice and repeatable. This function simply intakes a question and then does RAG to answer it.\n",
    "\n",
    "We start by embedding the question using the same embedding model we used to embed the documents. Next, we execute a search on Zilliz Cloud.\n",
    "\n",
    "Notice that we pass the question into the `get_embeddings` function in list format, and then pass the outputted list directly into the `data` section of our Zilliz Cloud search. This is because of the way that the function signatures are set up; it’s easier to reuse them than rewrite multiple functions.\n",
    "\n",
    "Inside our search call, we also need to provide a few more parameters. `anns_field` tells Zilliz Cloud which field to do an approximate nearest neighbor search (ANNS) on.\n",
    "\n",
    "Next, we also pass a `limit` parameter which tells us how many results to get back from Zilliz Cloud. For this example, we can just go with five.\n",
    "\n",
    "The last search parameter defines which fields we want back from our search. For this example, we can just get the `sentence`, which is the field we used to store our chunk of text.\n",
    "\n",
    "Once we have our search results back, we need to process them. Zilliz Cloud returns an entity with `hits` in it, so we grab the “sentence” from all five hits and join them with a period so it forms a list paragraph.\n",
    "\n",
    "Then, we pass the question that the user asked along with the context into the `dorag` function we created above and return the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9a9588bde8cbb838",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T20:08:13.589578Z",
     "start_time": "2024-04-22T20:08:13.213484Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 448985292834734942, 'distance': 0.8144542574882507, 'entity': {'sentence': 'Cambridge is located in eastern Massachusetts, bordered by:'}}\n",
      "{'id': 448985292834734969, 'distance': 0.7613034248352051, 'entity': {'sentence': 'Areas of Cambridge'}}\n",
      "{'id': 448985292834735059, 'distance': 0.7230358123779297, 'entity': {'sentence': 'Cambridge College is named for Cambridge and was based in Cambridge until 2017, when it consolidated to a new headquarters in neighboring Boston.'}}\n",
      "{'id': 448985292834735065, 'distance': 0.6981460452079773, 'entity': {'sentence': 'Cambridgeport School'}}\n",
      "{'id': 448985292834735141, 'distance': 0.6944277882575989, 'entity': {'sentence': 'Cambridge, Massachusetts at Curlie'}}\n",
      "Cambridge is located in eastern Massachusetts, bordered by:. Areas of Cambridge. Cambridge College is named for Cambridge and was based in Cambridge until 2017, when it consolidated to a new headquarters in neighboring Boston.. Cambridgeport School. Cambridge, Massachusetts at Curlie\n"
     ]
    }
   ],
   "source": [
    "question = \"What state is Cambridge in?\"\n",
    "\n",
    "def ask_a_question(question):\n",
    "   embeddings = get_embeddings([question])\n",
    "   res = zilliz_client.search(\n",
    "       collection_name=COLLECTION_NAME,\n",
    "       data=embeddings,  # search for the one (1) embedding returned as a list of lists\n",
    "       anns_field=\"embedding\",  # Search across embeddings\n",
    "       limit = 5,  # get me the top 5 results\n",
    "       output_fields=[\"sentence\"]  # get the sentence/chunk and city\n",
    "   )\n",
    "\n",
    "   sentences = []\n",
    "   for hits in res:\n",
    "       for hit in hits:\n",
    "           print(hit)\n",
    "           sentences.append(hit['entity']['sentence'])\n",
    "   context = \". \".join(sentences)\n",
    "   return context\n",
    "\n",
    "context = ask_a_question(question=question)\n",
    "print(context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6b8aa368d1b7571d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-04-22T20:20:47.772574Z",
     "start_time": "2024-04-22T20:20:38.907871Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Hello! I'm here to help you with your question. Based on the context provided, Cambridge is located in the state of Massachusetts. Specifically, it is situated in eastern Massachusetts, bordered by:\n",
      "\n",
      "* Boston to the south\n",
      "* Somerville to the west\n",
      "* Arlington to the north\n",
      "* Lexington to the northwest\n",
      "* Belmont to the west\n",
      "\n",
      "Cambridge is home to Cambridge College, which was named for Cambridge and was based in Cambridge until 2017, when it consolidated to a new headquarters in neighboring Boston. Additionally, Cambridgeport School is located in Cambridge.\n",
      "\n",
      "I hope this information helps answer your question! If you have any further queries, please feel free to ask.\n"
     ]
    }
   ],
   "source": [
    "print(dorag(question=question, context=context))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca6ac1aa138993e",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "For the example question asking which state Cambridge is in, we can print the entire response from BentoML. However, if we take the time to parse through it, it just looks nicer, and it should tell us that Cambridge is located in Massachusetts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dea0bfd6898d79c",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Summary: BentoML and Zilliz Cloud for RAG\n",
    "This example covered how you can do RAG without OpenAI or a framework. This time, our stack was the BentoML and Zilliz Cloud. We used BentoML’s serving capabilities to serve an embedding model endpoint and LLM endpoints to access an open source model and Zilliz Cloud as our vector database.\n",
    "\n",
    "There are many ways to structure the order in which we use these different puzzle pieces. For this example, we started this RAG project base on the cloud service from BentoML and Zilliz Cloud. We can actually start them locally as well.\n",
    "\n",
    "We used a simple method to chunk up our data, which was scraped from Wikipedia. Then, we took those chunks and passed them to our embedding model, hosted on BentoML, to get the vector embeddings to put into Zilliz Cloud. With all of the vector embeddings in Zilliz Cloud, we were fully set to do RAG.\n",
    "\n",
    "The LLM we chose this time was the `Llama-3-8B-Instruct` model, one of many open source models available on BentoML. We created two functions to enable RAG: one function that passed in the question and context to the LLM; and another function that embedded the user question, searched Zilliz Cloud and then passed in the search results along with the question to the original RAG function. At the end, we tested our RAG with a simple question as a sanity check."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
